exp/ticker: Orderbook data support #1193

accordeiro · 2019-04-26T15:55:25Z

This PR:

Closes exp/ticker: create an orderbook scraper to compute stats #1089

It aims to:

Create a script that scrapes orderbooks from Horizon to compute spreads and other stats
Incorporate orderbook data on the markets.json file that is constantly refreshed
Incorporate orderbook data on the GraphQL interface
Create a cron task to constantly refresh the orderbook data directly from Horizon

…sset types

accordeiro · 2019-04-29T16:25:04Z

Please wait until #1204 is merged before reviewing, since the tweaked db migrations for that PR will need to be merged here (and a new bindata.go file must be generated) before merging into master.

… tickerdb

accordeiro · 2019-04-29T19:40:52Z

Done! Safe to review now ✅

tomquisel · 2019-04-29T22:27:05Z

exp/ticker/internal/gql/static/schema.gql

@@ -74,7 +74,8 @@ type Market {
 	change: Float!
 	close: Float!
 	intervalStart: Time!
-	firstLedgerCloseTime: Time!
+	firstLedgerCloseTime: Time!,


I don't think you want this , or the ones below

Thanks! I've sent a fix for that.

tomquisel · 2019-04-29T22:28:38Z

exp/ticker/internal/main.go

@@ -32,6 +32,14 @@ type MarketStats struct {
 	Price            float64 `json:"price"`
 	Close            float64 `json:"close"`
 	CloseTime        int64   `json:"close_time"`
+	NumBids          int     `json:"num_bids"`


Thinking about it, maybe we should unify where Bid appears in these names? So maybe:

BidCount
BidVolume
BidHighest (or BidMax)

and the same for Ask variables

What do you think?

Yeah, that makes sense! I've pushed a change that updates all the related fields to have Bid or Ask in the beginning. I've also updated the GraphQL interface to reflect that.

tomquisel · 2019-04-29T22:31:47Z

exp/ticker/internal/scraper/orderbook_scraper.go

+	r := horizonclient.OrderBookRequest{}
+
+	switch bType {
+	case string(horizonclient.AssetTypeNative):


This seems a little verbose, is it possible to just assign type, code, and issuer in all cases?

Yes, I agree it was a little verbose. We still needed some logic here, as we store "XLM" as the asset code for native, and the Horizon API requires the asset code to be empty when type = native. I've updated the function and added some comments for context.

tomquisel · 2019-04-29T22:39:42Z

exp/ticker/internal/scraper/orderbook_scraper.go

+	if err != nil {
+		return obStats, errors.Wrap(err, "could not create a orderbook request")
+	}
+	summary, err := c.Client.OrderBook(r)


I believe it's possible to have multiple pages of offers, so I'd keep paging through if necessary.

I've taken a look at the Go SDK and client.OrderBook doesn't seem to return a Page-like entity. I've also checked the official Horizon documentation and there are no references to a _links field, so I'm guessing this specific endpoint isn't paginated. Is this some that was recently changed?

Hmm, you're right. That's a little unfortunate. There is a limit field, though, so it's definitely possible to make a request and get less than full orderbook data. Maybe just set the limit to 200 for now as a hack fix? Can you also add an issue for Horizon to add full pagination to the orderbook endpoint?

Done! I've just pushed a change that adds the 200 limit, and also created the Horizon issue #1230.

tomquisel · 2019-04-29T22:53:28Z

exp/ticker/internal/scraper/orderbook_scraper.go

+		if err != nil {
+			return errors.Wrap(err, "invalid bid price")
+		}
+		obStats.BidVolume += pricef


I think this should be obStats.BidVolume += bid.Amount

tomquisel · 2019-04-29T22:53:56Z

exp/ticker/internal/scraper/orderbook_scraper.go

+		obStats.HighestBid = 0
+	}
+	for _, bid := range summary.Bids {
+		pricef, err := strconv.ParseFloat(bid.Price, 64)


I'd grab the price from bid.PriceR instead, it'll be higher precision.

tomquisel · 2019-04-29T22:54:22Z

exp/ticker/internal/scraper/orderbook_scraper.go

+		obStats.LowestAsk = 0
+	}
+	for _, ask := range summary.Asks {
+		pricef, err := strconv.ParseFloat(ask.Price, 64)


Use ask.PriceR

tomquisel · 2019-04-29T22:55:01Z

exp/ticker/internal/scraper/orderbook_scraper.go

+		if err != nil {
+			return errors.Wrap(err, "invalid ask price")
+		}
+		obStats.AskVolume += pricef


I think this should be obStats.AskVolume += pricef * ask.Amount

Done! The fact that we're multiplying by the amount is related to #612, right?

exactly, yep! (well, the fact that we're multiplying by price).

tomquisel · 2019-04-29T23:01:33Z

exp/ticker/internal/tickerdb/queries_market.go

@@ -112,7 +129,14 @@ SELECT
 	COALESCE(open_price_7d, 0.0) as open_price_7d,

 	COALESCE(last_price, 0.0) as last_price,
-	COALESCE(last_close_time, now()) as close_time
+	COALESCE(last_close_time, now()) as close_time,


Hmm, I'm not sure this is the right fallback if there are no close times. Could this ever happen? Maybe it should be 0 instead?

Since it's a right join, there could be null 24h close times. I agree that now() didn't make sense, so I've updated it to fall back to the 7d close time.

tomquisel · 2019-04-29T23:04:32Z

exp/ticker/internal/utils/main.go

+	if askMin == 0 || bidMax == 0 {
+		return 0, 0
+	}
+	spread = math.Abs(askMin-bidMax) / askMin


You can remove the math.Abs, a negative spread is impossible in Stellar. But if it did happen, it'd be nice to know about it anyway since a negative spread is well defined.

tomquisel

Nice! Maybe I missed it, where's the logic that defaults 24h OHLC to the 7d close?

accordeiro · 2019-05-02T18:26:44Z

Just to document here – we've discussed this offline and I'll address the OHLC values issue (#1218) on a separate PR.

* horizonclient/txnbuild README fixes (#1210) * fix client links in top-level readme * update clients README * mark old client deprecated in godoc short summary * fix code of conduct, standardise example * fix code of conduct/contributing links * txnbuild: enables multiple signatures (#1198) This PR enables multiple signatures on a transaction in the new Go SDK. It also lets every `Operation` type have a different source account than its `Transaction`. These changes are intertwined. Without multiple signatures, every operation in a transaction must share the transaction's source account. Differing source accounts are the most common use case for multiple signatures, and they also test it with greatest completeness. * root repo changelog links to sub-projects (#1214) * keystore: add update-keys endpoints to spec (#1215) We need an endpoint to update the encrypted seed when users forget their passwords. * move keystore to exp to fix build (#1223) * Add minimal files to fix build (#1225) * exp/ticker: Orderbook data support (#1193) * exp/ticker: update partial market query to also output base/counter asset types * exp/ticker: add functions to scrape orderbook data * exp/ticker: update orderbookstats param names to match market standards * exp/ticker: create orderbookstats database model and migrations * exp/ticker: consolidate upsert logic and implement orderbookstats upsert * exp/ticker: create CLI command to ingest orderbooks * exp/ticker: update horizon SDK path * exp/ticker: add orderbook stats to market.json generator * exp/ticker: add separate query to retrieve relevant markets * exp/ticker: add orderbook stats to graphql interface * exp/ticker: add Docker support for orderbook ingestion * exp/ticker: fix partial aggregated market query + add tests * exp/ticker: fix globall aggregated market query + add tests * exp/ticker: update ticker binary link and fix crontab comment * exp/ticker: fix issue that would enable inf values to be stored in db * exp/ticker: ensure only store valid orderbook entries are stored * exp/ticker: format gql/static/bindata.go to prevent CI errors * exp/ticker: add updated bindata with pg 9.5-compatible migrations * exp/ticker: ensure markets with 0 orderbook entries can be handled by tickerdb * exp/ticker: remove unnecessary commas from graphql schema * exp/ticker: consolidate bid and ask positions on field names * exp/ticker: simplify logic for creating orderbook requests * exp/ticker: fix how bid and ask volumes are calculated * exp/ticker: allow negative spread values * exp/ticker: fix close_time fallback on markets query * exp/ticker: add page size limit to orderbook requests * Make PR template a bit more helpful. (#1238) * Make PR template a bit more helpful. Clean up some typos, and point to an example docs folder in the go repo rather than just the external docs hosted on our site. * Fix typo. * Agh, formatting. * horizonclient: update Fund method (#1213) * update client.Fund * update change log * fix space * horizonclient: add more documentation (#1226) * Remove old stream method * more comments * fix format * split comment * horizonclient: Set default HTTP client (#1228) * set default HTTP client * changes from review * changelog for txnbuild 1.1 (#1231) * horizonclient/txnbuild README fixes (#1210) * fix client links in top-level readme * update clients README * mark old client deprecated in godoc short summary * fix code of conduct, standardise example * fix code of conduct/contributing links * txnbuild: enables multiple signatures (#1198) This PR enables multiple signatures on a transaction in the new Go SDK. It also lets every `Operation` type have a different source account than its `Transaction`. These changes are intertwined. Without multiple signatures, every operation in a transaction must share the transaction's source account. Differing source accounts are the most common use case for multiple signatures, and they also test it with greatest completeness. * root repo changelog links to sub-projects (#1214) * keystore: add update-keys endpoints to spec (#1215) We need an endpoint to update the encrypted seed when users forget their passwords. * move keystore to exp to fix build (#1223) * Add minimal files to fix build (#1225) * changelog for txnbuild 1.1 * update change log (#1233) * horizonclient: add version (#1229) * add package version * fix go fmt * clients/horizonclient: support dynamic effects (#1217) * update effects in protocols/horizon * update client interface * update tests * update changelogs * fix typo

accordeiro added 17 commits April 24, 2019 18:54

exp/ticker: update partial market query to also output base/counter a…

5b37560

…sset types

exp/ticker: add functions to scrape orderbook data

6e1346c

exp/ticker: update orderbookstats param names to match market standards

5df5153

exp/ticker: create orderbookstats database model and migrations

44463a9

exp/ticker: consolidate upsert logic and implement orderbookstats upsert

3d0e4d2

exp/ticker: create CLI command to ingest orderbooks

bc5ff85

Merge branch 'master' into exp/ticker/orderbook

1eda6dc

exp/ticker: update horizon SDK path

fd0250a

exp/ticker: add orderbook stats to market.json generator

c3a1a78

exp/ticker: add separate query to retrieve relevant markets

5215bdb

exp/ticker: add orderbook stats to graphql interface

b25240e

exp/ticker: add Docker support for orderbook ingestion

d5fce2a

exp/ticker: fix partial aggregated market query + add tests

aa513d9

exp/ticker: fix globall aggregated market query + add tests

22a01aa

exp/ticker: update ticker binary link and fix crontab comment

c90b270

exp/ticker: fix issue that would enable inf values to be stored in db

3bc5ad4

exp/ticker: ensure only store valid orderbook entries are stored

f311ee4

accordeiro added the ticker label Apr 26, 2019

accordeiro requested a review from tomquisel April 26, 2019 15:55

accordeiro self-assigned this Apr 26, 2019

exp/ticker: format gql/static/bindata.go to prevent CI errors

9185656

accordeiro added 3 commits April 29, 2019 16:17

Merge branch 'master' into exp/ticker/orderbook

3f7c748

exp/ticker: add updated bindata with pg 9.5-compatible migrations

e534072

exp/ticker: ensure markets with 0 orderbook entries can be handled by…

4d24aa5

… tickerdb

tomquisel approved these changes Apr 29, 2019

View reviewed changes

accordeiro added 3 commits May 1, 2019 10:42

exp/ticker: remove unnecessary commas from graphql schema

9025afb

exp/ticker: consolidate bid and ask positions on field names

878d795

exp/ticker: simplify logic for creating orderbook requests

a1e9383

accordeiro added 3 commits May 1, 2019 13:27

exp/ticker: fix how bid and ask volumes are calculated

fbdff21

exp/ticker: allow negative spread values

f290fc8

exp/ticker: fix close_time fallback on markets query

37beb42

tomquisel reviewed May 2, 2019

View reviewed changes

exp/ticker: add page size limit to orderbook requests

361f5e1

accordeiro mentioned this pull request May 2, 2019

services/horizon: Add full pagination to Orderbooks endpoint #1230

Open

accordeiro merged commit 58904f9 into stellar:master May 2, 2019

accordeiro deleted the exp/ticker/orderbook branch May 2, 2019 18:37

leighmcculloch mentioned this pull request Jan 13, 2021

Odd choice of returned units from orderbookCallBuilder.call() in stellar-sdk #612

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

exp/ticker: Orderbook data support #1193

exp/ticker: Orderbook data support #1193

accordeiro commented Apr 26, 2019

accordeiro commented Apr 29, 2019 •

edited

Loading

accordeiro commented Apr 29, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019 •

edited

Loading

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel May 2, 2019

accordeiro May 2, 2019 •

edited

Loading

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel May 2, 2019

tomquisel Apr 29, 2019

accordeiro May 1, 2019 •

edited

Loading

tomquisel Apr 29, 2019

accordeiro May 1, 2019

tomquisel left a comment

accordeiro commented May 2, 2019 •

edited

Loading

exp/ticker: Orderbook data support #1193

exp/ticker: Orderbook data support #1193

Conversation

accordeiro commented Apr 26, 2019

accordeiro commented Apr 29, 2019 • edited Loading

accordeiro commented Apr 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

accordeiro May 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

accordeiro May 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

accordeiro May 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tomquisel left a comment

Choose a reason for hiding this comment

accordeiro commented May 2, 2019 • edited Loading

accordeiro commented Apr 29, 2019 •

edited

Loading

accordeiro May 1, 2019 •

edited

Loading

accordeiro May 2, 2019 •

edited

Loading

accordeiro May 1, 2019 •

edited

Loading

accordeiro commented May 2, 2019 •

edited

Loading